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FOREWORD 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  conducts 
research  on  how  to  design  unit  training  strategies.  Within  the  past  few  years,  its  mission  has  been 
extended  to  include  the  Army’s  role  in  inter  service  training.  This  document  is  one  of  a  series  of 
reports  that  have  been  produced  under  the  Joint  and  Multi-Service  Distributed  Training  Testbed 
(JMDT2)  program.  The  purpose  of  the  program  has  been  to  apply  the  basic  concepts  of 
instructional  systems  design  (ISD)  to  the  development  and  management  of  inter  service  training. 

A  critical  feature  of  ISD  is  assessment  of  training  progress  and  remedial  feedback. 

This  report  summarizes  the  beginning  of  innovative  efforts  to  introduce  ground  breaking 
new  ideas  about  training  feedback.  These  ideas  are  extensions  of  classical  ISD  concepts, 
especially  tailored  to  the  unstructured,  non-linear,  often  fuzzy  conditions  of  high  echelon  inter 
service  mission  planning  and  management.  The  work  extends  the  Army’s  Socratic  after  action 
review  (AAR)  method  of  feedback  by  adding  the  philosophy  and  methods  of  the  Delphi 
Approach.  The  trainee  is  recognized  as  an  untapped  source  of  a  wealth  of  information  about 
training  deficiencies  and  training  progress,  particularly  in  inter  service  training.  Methods  for 
extracting  that  information  according  to  Delphi  procedures  were  developed  and  then  pilot  tested 
in  a  division  artillery  staff  training  exercise  at  Fort  Hood,  in  February,  1998.  Preliminary  results  of 
an  assessment  of  user  reactions  indicate  that  the  methodology  can  substantially  leverage  the 
effectiveness  of  future  staff  training  at  high  echelons,  whether  single  or  inter  service. 


zA  M.  SIMUTIS 
Tfechnical  Director 
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SELF-ASSESSMENT  BASED  MINI- AAR  (SAMAAR)  METHODOLOGY; 
DEVLOPMENTAL  RESEARCH  IN  DIVISION  ARTILLERY  STAFF  TRAINING 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  overall  requirement  was  to  pilot  test  the  Self-Assessment  Based  Mini- After  Action  Review 
(SAMAAR)  methodology.  Specific  goals  included  the  following:  (1)  Assess  the  value  and 
features  of  SAMAAR  tools;  (2)  Assess  problems  in  research  Implementation  of  SAMAAR;  and 
(3)  Compile  “Lessons  Learned”  for  follow-up  Corps-level  Research  Application  of  SAMAAR. 

Procedure; 

The  research  test  bed  was  provided  by  a  regularly  scheduled  field  training  exercise  (FTX)  of  the 
1®*  Cavalry  Division  Artillery  Staff  at  Fort  Hood,  TX.  Research  participants  included  members  of 
three  staff  elements:  Operations  (Ops),  Fire  Control  Element  (FCE),  and  the  Division  Tactical 
Center  Fire  Support  Element  (DTAC  FSE).  A  self-assessment  checklist  was  distributed  to  staff 
members  at  the  beginning  of  each  of  two  days  of  a  field  training  exercise,  along  with  some  job 
aids  describing  the  tasks  for  each  staff  element.  It  was  anticipated  that  filling  out  the  checklist 
forms  would  be  a  preliminary  step  in  preparation  for  an  end  of  shift  mini- AAR  on  each  day.  For  a 
variety  of  administrative  reasons  the  mini-AARs  were  not  conducted,  but  the  checklists  were 
completed  by  staff  members.  At  the  end  of  two  exercise  days,  a  questionnaire  to  assess  the  value 
of  the  self-assessment  approach  and  tools  was  administered. 

Findings: 

The  SAMAAR  approach  was  judged  to  be  useful  and  timely  for  surfacing  training  problems  that 
otherwise  might  be  overlooked.  The  self-assessment  checklist  was  judged  to  be  clear  and  easy  to 
complete.  It  was  demonstrated  how  checklist  data  under  standardized  conditions  can  provide  a 
way  to  track  training  progress  across  unit  elements  and  mission  days.  The  value  and  appropriate 
use  of  the  job  aids  -  task  flow  charts  and  tables  -  are  inconclusive.  The  job  aids  require  further 
development  and  test. 

Use  of  Findings: 

The  findings  provide  a  technical  base  for  further  research  and  development  of  the  self-assessment 
approach  in  a  joint  training  environment.  Lessons  were  learned  about  how  to  improve  the 
implementation  of  the  self-assessment  methodology  and  how  to  conduct  research  on  its 
effectiveness. 
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Self-Assessment  Based  Mini-After  Action  Review  (SAMAAR)  Methodology: 
Developmental  Application  to  Division  Artillery  Staff  Training 

INTRODUCTION 


Program  Background 

This  report  describes  research  conducted  as  part  of  an  ARI  program  to  develop  measures 
and  methods  for  supporting  after  action  reviews  (AARs)  in  joint  and  multi-service  training.  The 
purpose  of  the  report  is  to  summarize  developmental  research  on  a  Self-Assessment  Based,  Mini- 
After  Action  Review  (SAMAAR)  methodology  that  has  evolved  from  earlier  efforts  by  and 
under  the  auspices  of  the  US  Army  Research  Institute  (ARI)  (Love,  1998a,  1998b,  1998c).  The 
current  research  was  conducted  as  a  supplement  to  a  Division  Artillery  (DivArty)  Staff  Training 
Event  at  Fort  Hood,  in  January  1998.  It  resulted  in  “lessons  learned”  for  follow-on  research 
dealing  with  joint  training. 

The  SAMAAR  methodology  is  designed  to  increase  the  efficiency  and  effectiveness  of 
training  feedback  and,  in  turn,  training  effectiveness.  Training  feedback  is  defined  here  as 
diagnostic  information  regarding  the  effectiveness  of  task  performance.  The  ARI  program 
focuses  on  feedback  generated  by  the  trainees  themselves  (i.e.,  “self-assessment”)  and 
incorporated  into  daily  staff  cell  AARs  (i.e.,  “mini”  AARs)  conducted  prior  to  unit-wide  AARs 
at  the  end  of  exercises  (EndExs). 

The  Army  uses  a  standardized  form  of  AAR  which  draws  heavily  on  the  trainees’ 
assessments  of  their  units’  strengths  and  needs  for  improvement  (Brown,  Wilkinson,  Nordyke, 
Riede,  Huyssoon,  Aguilar,  &  Wonsewitz,  &  Meliza,  1997,  p.  3;  Department  of  the  Army,  1993; 
Dyer,  1994;  Keene,  1994;  Meliza,  1996;  &  Scott,  1984).  This  form  is,  in  fact,  an  application  of 
the  Socratic  method.  In  a  typical  Army  AAR,  a  senior  observer/controller  (0/C)  moderates  a 
discussion  of  what  happened,  how  and  why  it  happened,  and  how  unit  performance  can  be 
improved  in  subsequent  exercises  or  exercise  phases.  Consistent  with  the  Socratic  method,  the 
Senior  0/C  is  likely  to  steer  the  discussion  towards  particular  problems  which  he  and  his  cadre  of 
0/Cs  have  noted  and  documented  during  the  training  exercise.  But  he  will  through  questioning 
draw  out  from  the  trainees  descriptions  of  their  own  experiences  and  thought  processes. 

The  AAR  methodology  is  based  on  the  assumption  that  active  involvement  of  the  trainee 
yields  motivational  and  learning  benefits.  Furthermore,  the  trainee  provides  critical  information 
and  judgments  about  training  effectiveness,  judgments  which  supplement  information  and 
assessments  made  by  0/Cs  (Scott,  1983,  pp.  2-5). 

This  AAR  methodology  has  been  exploited  and  institutionalized  extensively  for  tactical 
training  at  the  combat  training  centers  (CTCs)  where  complex  instrumentation  and  trained  cadres 
of  0/Cs  are  available  (Dyer,  1994;  Sulzen,  1997;  U.S.  Army  Research  Institute,  1992).  But  the 
methodology  is  less  thoroughly  used  in  the  joint  and  multi-service  arenas,  even  though  AAR 
principles  and  practices  are  explicitly  and  thoroughly  detailed  in  the  Joint  Training  Manual 
(Joint  Chiefs  of  Staff,  1996a).  “Hot  washes”  (i.e.,  short  summary  briefings  about  exercise  pluses 
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and  minuses  in  key  areas)  and  after-action  reports  do  provide  good  training  feedback  in  Theater 
and  Joint  Task  Force  (JTF)  level  training  events.  However,  these  occur  at  the  end  of  two  to  three 
week  rotations,  are  aimed  primarily  at  senior  leadership,  and  deal  as  much  with  training  system 
and  training  management  problems  as  with  staff  performance  problems.  Some  observer/trainers 
may  hold  an  AAR  following  an  exercise,  but  even  this  is  not  done  after  every  exercise.  The 
foregoing  is  not  a  criticism,  but  simply  an  observation  about  an  opportunity  to  leverage  joint 
training  effectiveness  at  relatively  little  cost. 

According  to  psychological  principles,  training  feedback  needs  to  be  done  frequently  and 
close  to  the  targeted  performance  to  be  effective  (Fedor  &  Buckley,  1987;  Weinstein,  Goetz,  & 
Alexander,  1988,  p.  122;).  Its  effectiveness  has  been  shown  to  decrease  in  an  inverse  exponential 
manner,  i.e.,  effectiveness  drops  sharply  at  first,  then  levels  off  (Klein,  1991,  p.  137;  Wolfle, 
1951).  “Theory”  also  recognizes  that  trainees  have  valid  and  valuable  insights  into  training 
progress  and  effectiveness  (Dominick,  Reilly,  &  McGourty,  1997;  Keesling,  Ford,  &  Harrison, 
1994;  Mirabella,  Sticha,  &  Morrison,  1997;  Zazanis  &  Lappin,  1998).  For  example,  evidence 
from  National  Training  Center  (NTC)  exercises  supports  the  validity  of  self-assessment  for 
battalion  task  force  training.  In  these  exercises  effectiveness  ratings  by  battalion  staff  persormel 
agreed  with  ratings  by  0/Cs  (Keesling,  Ford,  &  Harrison,  1994,  p.  145).  Zazanis  and  Lappin 
(1998)  studied  the  use  of  trainees  as  sources  of  evaluation  data  for  special  forces  training.  Their 
research  suggested  that  such  data  “...  may  provide  the  most  critical  and  irreplaceable  information 
in  situations  that  have  a  high  requirement  for  interpersonal  skills,  or  that  have  difficult  or 
ambiguous  tasks  requiring  high  levels  of  effort  or  persistence.”  Therefore,  a  wealth  of 
knowledge  may  remain  untapped  in  the  experiences  of  trainees,  knowledge  that  can  be  extracted 
through  enhanced  self-assessment  tools  and  methods.  The  characteristics  of  that  methodology 
are  described  below. 


The  SAMAAR  Approach 


Overview 

The  SAMAAR  approach  expands  the  philosophy  and  techniques  of  the  Army’s  Socratic 
AAR  method  by  adding  the  philosophy  and  techniques  of  the  Delphi  method.  The  Army’s 
approach  has  already  been  described.  The  Delphi  method  adds  a  mechanism  for  trainees  to 
individually  think  about  and  document  their  perceptions  of  unit  training  progress  at  their  level  of 
experience,  so  that  they  are  well  prepared  to  contribute  to  a  follow-up  mini-AAR.  The 
mechanism  is  designed  to  draw  out  participant  experience,  maximally  and  efficiently,  while 
correcting  limitations  in  a  given  trainee’s  situation  awareness. 

In  its  original  applications  Delphi  was  used  to  facilitate  the  efforts  of  experts  to  solve 
fuzzy,  complex  national  policy  problems.  The  experts  were  asked  to  generate  and  document 
solutions  individually  and  then  convene  for  group  discussion  to  reconcile  differences  and  arrive 
at  an  efficient  consensus.  In  getting  to  the  consensus,  differences  among  the  experts  were 
discussed.  These  might  be  the  result  of  different  premises  or  experiences.  They  were  factored 
into  the  consensus  solution.  Thus  Delphi  was  a  mechanism  for  increasing  the  reliability  of 
subjective  “measurement.” 
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An  adaptation  of  Delphi  philosophy  and  techniques  appears  well  suited  to  assessing  staff 
training  performance,  especially  the  very  “soft”  skills  inherent  in  operations  planning  and 
concept  development.  These  soft  skills  become  increasingly  important  at  increasing  echelon 
levels  and  are  prevalent  in  joint  training  environments. 

Measurement  Characteristics 


Trainee  as  the  Data  Generator.  Collective  training  effectiveness  measurement  has 
traditionally  employed  0/Cs,  electronic  data  streams,  or  both  as  data  sources  as  at  the  Army’s 
Combat  Training  Centers.  0/C  cadres  are  employed  as  data  generators  for:  (a)  ARI  s  Army 
Command  and  Control  Evaluation  System  (ACCES),  (b)  the  Naval  Air  Warfare  Center’s 
Tactically  Relevant  Assessment  of  Combat  Teams  (TRACT),  and  (c)  The  Tactically  Relevant 
Assessment  of  Combat  Events  (TRACE)  methodologies,  as  well  as  for  most  joint  training 
exercises.  The  SAMAAR  approach  is  somewhat  unique  in  viewing  the  trainee  as  a  surrogate 
0/C  and  as  a  key  source  of  training  assessment  data.  Therefore,  where  an  0/C  cadre  is  available 
it  would  be  leveraged  by  the  trainees.  Where  a  cadre  is  not  available,  the  trainees  would  act  a 
surrogate  cadre. 

Task  as  Unit  of  Analysis.  The  SAMAAR  methodology  is  task  oriented  rather  than 
events-based.  A  similar  orientation  can  be  seen  for  the  measurement  approach  in  the  Universal 
Joint  Task  List  (UJTL).  But  SAMAAR  contrasts  with  TRACE,  which  is  used  to  verify  the 
actual  occurrence  of  fine-grained,  sequenced  combat  events  against  a  list  of  expected  events. 
SAMAAR  is  more  consistent  with  a  view  of  battle  as  dynamically  nonlinear  and  complexly 
interactive.  TRACE  is  more  consistent  with  a  view  of  battle  as  a  predictable  sequence  of  events. 
Each  perspective  is  a  simplifying  assumption  since  combat  clearly  has  both  linear  and  nonlinear 
characteristics.  Nonetheless,  the  perspectives  are  not  necessarily  equally  useful  in  generating 
measurement  methodology  for  supporting  training  feedback.  A  forerunner  of  TRACE  was  found 
to  be  only  marginally  useful  in  supporting  AARs  in  a  battalion  task  force,  simulation-based 
training  event  (Mirabella,  Sticha,  &  Morrison,  1997).  SAMAAR,  on  the  other  hand,  has  evolved 
specifically  with  AARs  in  mind.  SAMAAR  also  contrasts  with  approaches  that  attempt  to 
measure  universal  (psychological)  teamwork  characteristics,  e.g.,  the  TRACT  methodology.  A 
forerunner  of  TRACT  was  also  examined  in  the  battalion  task  force  exercises.  Exercise  0/Cs 
were  unanimous  in  reporting  that  it  was  not  useful  for  preparing  or  conducting  AARs  (Mirabella, 
Sticha,  &  Morrison,  1997). 

Process  and  Product  Measurement.  SAMAAR  measures  behavioral  processes  as  well  as 
products  and  directly  assesses  perceived  training  deficiencies.  It  contrasts  with  ACCES  (Halpin, 
1996)  and  with  the  UJTL  (Joint  Chiefs  of  Staff,  1995,  1996a,b)  measurement  scheme.  ACCES 
and  UJTL  employ  “statistical”  dimensions  to  measure  the  outputs  of  process,  e.g.,  percentage, 
duration,  quantity,  and  distance.  This  “statistical”  approach  for  feedback  has  received  some 
negative  comment  {e.g.,  at  the  1997  Universal  Joint  Task  List  Conference,  Joint  Warfighting 
Center,  Fort  Monroe,  VA)  as  being  too  rigid,  restrictive,  and  fine-grained  for  the  “fuzzy” 
character  of  training  at  high  echelons.  SAMAAR  is  deliberately  more  discursive;  it  avoids  the 
emphasis  on  mathematical  precision  implicit  in  ACCES  and  UJTL. 
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Easy  Application.  SAMAAR  is  designed  for  frequent  application  (e.g.,  immediately 
following  the  end  of  each  shift).  It  calls  for  a  15-minute  period  at  the  end  of  a  day  or  shift  for 
trainees  to  fill  out  an  assessment  of  their  cell’s  performance,  using  a  short  checklist  about  key 
products  and  processes.  This  activity  is  followed  by  a  mini-AAR  which  is  led  by  the  senior 
member  of  a  staff  cell  (or  tactical  sub  unit).  Availability  of  an  0/C  cadre  is  not  assumed  for  use 
of  the  methodology.  AARs  that  rely  on  0/C  or  electronic  data  require  extensive  preparation  time 
and  personnel  resources  (Brown  et  al,  1997;  Meliza,  1996) 

Multiple  Uses.  The  SAMAAR  approach  is  designed  to  maximize  involveinent  of  all 
players  in  the  training  process  and  learning  experience.  First  it  provides  an  unconstrained 
opportunity  for  each  member  of  a  staff  cell  to  record  his  or  her  reactions  to  training  progress. 
Then  it  provides  for  a  meeting  in  which  each  staff  member’s  voice  has  equal  weight  in  arriving  at 
a  consensus  (or  disagreements)  on  where  training  improvements  are  needed. 

The  method  can  also  facilitate  hand-off  to  the  next  shift.  Problems  identified  by  staff 
cells  in  one  shift  can  be  highlighted  for  cells  in  the  following  shift.  Similarly  inter-cell  problems, 
e.g.,  communication  shortfalls,  can  be  addressed  for  the  benefit  of  that  and  subsequent  shifts. 

Finally,  results  of  the  SAMAAR  process  can  be  made  available  for  and  integrated  into 
unit-wide  AARs  at  the  end  of  the  training  event  or  rotation.  As  part  of  this,  the  results  of  the 
SAMAAR  process  could  be  used  to  do  a  trend  analysis  across  days  within  the  training  event  or 
across  training  events. 

Anticipated  Training  Benefits 

Based  upon  principles  of  learning  psychology,  we  can  anticipate  the  following  positive 
effects  of  the  SAMAAR  approach  on  training  progress; 

1 .  Timely  Feedback.  It  was  mentioned  earlier  that  the  effectiveness  of  training  feedback 
increases  the  closer  it  is  given  to  when  the  targeted  performance  occurs.  The  logistic  simplicity 
of  SAMAAR  makes  frequent  application  of  the  approach  feasible. 

2.  Increased  Time  on  Task  (TOT).  The  SAMAAR  approach  is  a  very  efficient  way  to 
amplify  the  effective  amount  of  practice  time  on  mission-essential  tasks,  because  in  half  an  hour 
or  so,  attention  can  be  focused  on  tasks  most  in  need  of  improvement.  This  can  be  viewed 
almost  as  a  form  of  compressed  training.  In  turn,  the  increased  TOT  can  substantially  raise  the 
unit’s  skill  level  and  decrease  the  rate  of  skill  decay  (Loftus,  1985;  Mirabella,  Macpherson,  & 
Patterson,  1989,  p.  40). 

3.  Active  Involvement.  The  method  increases  active  involvement  by  encouraging  inputs 
to  the  feedback  process  from  each  member  of  a  staff  cell  or  other  sub-unit  involved  in  mini- 
AARs.  Active  participation  is  critical  to  effective  training  (Mirabella,  Macpherson,  &  Patterson, 
1989;  Travers,  Van  Wagenen,  Haygood,  &  McCormick,  1964). 
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SAMAAR  Procedures 


Orientation 


The  purpose  of  SAMAAR  and  its  procedures  need  to  be  clearly  explained  to  all  the 
trainees  who  will  be  asked  to  fill  in  the  self-assessment  checklists  and  take  part  in  the  mini- 
AARs.  Orientation  should  not  be  limited  to  the  staff  cell  leaders.  The  orientation  should  stress 
that  SAMAAR  is  designed  to  encourage  all  the  trainees  to  participate  in  and  contribute  to  the 
AAR  process.  But  it’s  meant  to  provide  a  set  of  tools,  not  a  “straight  jacket.”  It  also  needs  to  be 
clear  that  filling  out  the  checklist  is  a  way  for  the  trainees  to  record  their  opinions  about  training 
progress  so  that  they  can  contribute  to  the  follow-up  mini-AAR.  Finally,  the  SAMAAR 
orientation  should  be  included  with  other  pre-exercise  preparations,  so  that  it  becomes  an  integral 
part  of  the  training,  rather  than  a  fringe  activity. 

Administration  of  Self-Assessment  Checklist 

At  the  end  of  a  shift,  trainees  should  be  gathered  in  a  setting  as  comfortable  as 
circumstances  will  permit.  Provide  a  table  if  possible  and  offer  pencils.  Go  over  the  highlights 
of  the  initial  orientation.  Then  explain  the  different  parts  of  the  checklist  and  the  rating 
procedure,  e.g.,  explain  what  the  checklist  categories  mean.  Pick  out  a  few  items  and  illustrate 
some  possible  reactions  to  those  items.  Encourage  the  participants  to  add  comments,  especially 
where  trainees  indicate  a  need  for  improvement.  Finally,  ask  for  questions  and  have  the  trainees 
complete  the  self-assessment  checklist. 

Conduct  of  the  Mini-AAR 


“House  Rules” 


•  Everyone  has  an  equal  voice  in  discussion;  no  one  has  all  the  answers.  Listening  and  talking 
are  equally  important  to  good  training  feedback. 

•  Focus  on  what  happened,  what  needs  improvement,  and  how  to  improve. 

•  Use  the  mini-AAR  to  spot  and  clear  up  misunderstandings  about  mission  process  or  product, 
especially  for  new  staff  members. 

Procedures  for  Conducting  “Self-Assessment”  Mini-AARs 

•  Staff  cell  gets  together  at  the  end  of  each  shift  for  a  mini-AAR.  Staff  members  bring  their 
filled-out  checklists  to  help  them  contribute  to  the  mini-AAR. 

•  Senior  cell  member  leads  the  mini-AAR,  starting  with  a  blank  checklist 

•  Steps  for  conducting  the  mini-AAR. 

a.  Address  each  checklist  item  in  turn. 

b.  If  everyone  agrees  to  a  satisfactory  rating,  go  to  the  next  item. 
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c.  Otherwise,  discuss  the  item  and  make  notes  on  training  problems  and  solutions. 

d.  If  a  member  of  the  cell  (e.g.,  new  replacement)  doesn’t  understand  a  mission  product 
or  process,  use  this  opportunity  to  clear  up  questions.  Then: 

>  AAR  leader  records  a  consensus  response. 

>  If  no  consensus,  leader  picks  a  response,  but  makes  notes  on  disagreements. 

e.  Keep  the  consensus  sheet  for  use  in  shift  change  over,  to  check  progress  over 
subsequent  shifts  or  mission  phases,  and  as  an  input  to  a  larger  AAR  at  the  end  of  the  training 
rotation. 


Research  Goals 


Overall  Goal 


Pilot  Research  Application  of  SAMAAR  Approach.  The  primary  audience  for  SAMAAR 
is  the  joint  training  community.  However,  the  division  artillery  exercises  at  Fort  Hood  provided 
a  target  of  opportunity  and  a  credible  surrogate  environment  to  conduct  pilot  research  in 
preparation  for  an  anticipated  joint  training  research  application.  But,  in  addition,  the  results  of 
the  pilot  application  were  expected  to  extend  the  Army’s  technical  base  on  training  feedback. 

Specific  Objectives 

1 .  Assessment  of  Value  and  Features  of  SAMAAR  Tools.  A  major  goal  was  to  assess 
user  reactions  to  the  following  questions:  Does  the  SAMAAR  approach  and  its  specific  tools 
add  value  to  staff  training  exercises?  What  improvements  or  revisions  are  needed? 

2.  Assessment  of  Problems  in  Research  Implementation  of  SAMAAR.  Since  this  was  a 
pilot  effort  to  support  follow-on  research  planning,  an  important  objective  was  to  document 
problems  in  scheduling  and  administering  SAMAAR  and  research  procedures.  How  well  could 
these  be  incorporated  into  the  on-going  exercises,  given  the  constraint  of  non-interference?  What 
difficulties  would  the  units  experience  in  using  the  self-assessment  checklist  to  conduct  mini- 
AARs? 


3.  Compilation  of  “Lessons  Learned”  for  Corps-Level  Research  Application  of 
SAMAAR.  This  objective  was  to  draw  on  our  division  Artillery  experience  to  design  SAMAAR 
and  write  a  research  plan  for  follow-on  research  in  a  joint,  corps-level  exercise  test  bed. 
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METHOD 


Test  Bed 

The  research  test  bed  was  provided  by  a  regularly  scheduled  field  training  exercise  (FTX) 
of  the  1st  Cavalry  Division  Artillery  Staff  at  Fort  Hood,  TX.  This  was  a  three-day  exercise  from 
20  to  22  January  1998.  One  shift  of  trainees  participated  on  Days  1  and  3;  a  second  shift 
participated  on  Day  2.  The  FTX  differed  in  one  important  respect  from  other  rotations.  It 
provided  a  primary  test  bed  for  research  by  the  Fort  Sill  Field  Unit  of  the  Army  Research 
Laboratory  (ARL).  ARI  worked  under  the  auspices  of  ARL,  on  the  basis  of  non-interference 
with  either  the  exercise  or  the  ARL  research  activities.  ARL’s  purpose  was  to  examine  the 
training  effectiveness  of  a  simulator/stimulator  developed  for  operational  testing,  the  Fire 
Support  Automated  Test  System  (FSATS).  A  related  purpose  was  to  test  combat  team 
measurement  instruments  adapted  from  the  Mult-Service  Distributed  Training  Test  Bed  (MDT2) 
program  (Bell,  Dwyer,  Love,  Meliza,  Mirabella,  &  Moses,  1997). 

Participants 

Research  participants  included  members  of  four  staff  elements:  Operations  (OPS),  Fire 
Control  Element  (FCE),  Targeting  Element  (TGTG),  and  the  Division  Tactical  Operations 
Center  Fire  Support  Element  (DTAC  FSE).  Appendix  A  lists  the  positions  in  each  of  the 
elements  which  did  participate.  Two  shifts  of  staff  members  went  through  training.  The  first 
shift  trained  on  Days  1  and  3,  the  second  on  Day  2.  But  some  staff  members  participated  in  both 
shifts. 

Research  Materials 


SAM  AAR  tools:  Package  with  introduction,  self-assessment  checklist  for  the  staff 
element,  flow  chart  showing  inputs  from  and  outputs  to  other  staff  elements,  table  listing  inputs, 
outputs,  and  processes  that  link/convert  inputs  to  outputs.  These  packages  were  compiled  for 
each  staff  element  from  the  set  of  materials  shown  in  Appendix  B. 

Research  tools 


SAMAAR  opinion  questionnaire.  A  questionnaire  was  designed  to  assess  the 
participants’  reactions  to  the  value  and  features  of  the  SAMAAR  approach  and  its  tools 
(Appendix  C).  The  questionnaire  contained  five  items  on  value  and  five  items  on  the  features  of 
the  SAMAAR  approach  and  tools.  A  six  point  response  scale  ranging  from  strongly  agree  to 
strongly  disagree  was  included  with  each  questionnaire  item.  The  scale  was  patterned  after  one 
used  by  Mirabella,  Sticha,  and  Morrison  (1997).  The  survey  form  also  provides  space  for 
comments. 
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Tape  recorders.  Tape  recorders  were  brought  to  the  test  site  to  record  the  mini-AARs. 
We  intended  to  content-analyze  the  recordings  for  problems  with  use  of  the  self-assessment 
forms  during  the  mini-AARs.  These  recordings  were  to  supplement  observations  of  the  AARs 
by  the  research  staff  following  a  practice  employed  by  Mirabella,  Sticha,  and  Morrison  (1997) 
in  the  MDT2  program.  The  tape  recorders  were  not  used  for  reasons  indicated  below. 

Procedures 


Orientation  to  SAMAAR.  On  the  morning  of  Day  1  a  series  of  orientation  briefings  were 
given  to  the  entire  DivArty  staff  by  the  ARL  research  staff.  Included  among  the  briefings  was  a 
short  (approximately  5-minute)  overview  of  and  introduction  by  ARI  to  the  SAMAAR  approach. 

Administration  of  self  assessment  checklists.  On  Days  1  and  2,  sets  of  SAMAAR 
packages  (Appendix  B)  were  given  to  each  of  the  DivArty  staff  element  non-commissioned 
officers  in  charge  (NCOICs).  These  were  to  be  distributed  to  each  member  of  each  of  the  four 
staff  elements.  The  assessment  checklist  in  the  package  was  to  be  completed  at  the  end  of  the 
day  (shift),  prior  to  a  mini-AARs.  Checklists  were  not  distributed  on  Day  3  because  we  learned 
that  the  exercise  was  to  be  terminated  after  a  few  hours  and  time  for  administering  either  the  self- 
assessment  checklists  or  the  mini-AARs  was  not  anticipated. 

Mini-AARs.  For  administrative  reasons  beyond  our  control,  the  expected  mini-AARs  on 
Days  1  and  2  were  not  conducted.  Consequently,  we  were  able  to  apply  only  the  first  segment  of 
the  SAMAAR  approach. 

Administration  of  opinion  survey  on  SAMAAR.  The  opinion  survey  (Appendix  C)  was 
administered  on  the  morning  of  Day  3,  as  part  of  a  larger  administration  of  questionnaires  by  the 
ARL  staff. 


RESULTS 


Opinions  on  Value  and  Features  of  SAMAAR 

Responses  to  the  evaluation  questionnaire  were  received  from  8  participants.  A  summary 
of  responses  to  a  reduced  set  of  items  is  presented  in  Table  1 .  A  munber  of  the  items  on  the 
original  questionnaire  dealt  with  mini-AARs,  which  never  took  place.  As  a  result  checklist 
responses  to  those  items  were  considered  uninterpretable.  Table  1  lists  the  remaining  items  and 
their  percentages  of  agreement  (“Agree”  to  “Strongly  Agree”).  Positive  reaction  is  uniformly 
high  for  value  and  features  (75%  to  100%  agreement). 


Table  1.  Evaluation  of  SAMAAR 


Evaluation  Item 

Agree 

Need/value  .  • 

4.  A  self-assessment  package  like  this  is  not  currently  available  to  my  unit 

100 

5.  A  self-assessment  package  like  this  or  an  improved  version  would  be  useful  in 
future  exercises 

88 

;FeaturesJ- ‘4.  ' 

6.  The  instruction  sheet  was  clear,  well  organized,  and  had  about  the  right 
amount  of  information. 

75 

7.  Assessment  checklist  items  were  useful  as  memory  joggers  and  could  be 
answered. 

88 

10.  I  felt  that  I  could  respond  accurately  to  the  self-assessment  tools 

75 

Comments  from  the  questionnaires  were  compiled  and  organized  from  the  entire  original 
questionnaire.  For  developmental  purposes,  it  was  considered  useful  to  examine  all  the 
comments.  These  are  clustered,  with  annotations,  under  a  set  of  categories  derived  from  the 
original  questionnaire.  Included  are  positive  eind  negative  comments  on  the  value  of  the  mini- 
AAR  methodology.  Also  included  are  constructive  comments  on  technical  aspects  of  the 
methodology.  Each  category  of  comment  provides  useful  information  and  “lessons  learned”  for 
further  research  and  development,  and  eventual  implementation  of  the  methodology. 

Positive  Comments 


The  positive  comments  indicate  that  the  self-assessment  approach  can  be  useful  and 
timely  in  surfacing  training  problems  that  might  have  otherwise  been  overlooked.  It  is  also  a 
way  to  ensure  that  all  levels,  enlisted  persormel  and  officers,  participate  in  improving  the 
training,  and  that  communication  and  coordination  problems  among  cells  not  “fall  through  the 
cracks.”  This  conclusion  is  based  on  the  following  comments: 

“More  detailed  AAR  package  would  be  a  great  tool  to  help  improve  our  performance.” 
“Although  not  a  replacement  for  the  formal  AAR,  the  mini-AAR  enhances  training 
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through  self-evaluation.” 

“It  is  a  good  tool  for  leaders  to  look  at  the  way  they  do  things  and  make  quick 

improvements.” 

“If  utilized  correctly,  the  mini-AAR  has  value.  It  prompted  me  to  look  at  areas  I  might 
have  overlooked.” 

“It  made  us  aware  of  what  to  give  a  little  more  attention  to  as  we  went  along.” 

“It  helped  identify  areas  where  improvement  is  needed  as  well  as  being  able  to  coordinate 
with  the  other  sections.” 

“Gets  input  from  all  levels.” 

“Have  them  at  the  end  of  the  exercise.” 

The  positive  comments  suggest  that  in  our  future  explanations  and  training  we  should 
point  out  that  the  mini-AAR  can  provide  “just-in-time”  remedial  feedback  —  feedback  which  is 
given  at  least  at  the  end  of  each  training  day,  in  small  enough  groups,  so  that  all  members  of  a 
staff  cell  can  contribute.  A  number  of  comments  from  an  ARL  survey  indicate  interest  in  having 
AARs  several  times  a  day  (Linda  Pierce,  personal  communication,  February,  1998).  One  trainee 
in  the  ARL  survey  suggested  AARs  at  the  end  of  one-hour  blocks,  a  second  trainee  suggested 
AARs  at  the  end  of  four-hour  blocks.  We  also  need  to  point  out  that  the  mini-AAR  can  help  to 
point  up  problems  in  inter-cell  communication  and  coordination.  These  can  then  be  worked  out 
either  informally  among  cells  or  as  part  of  a  larger  AAR  that  brings  the  cells  together. 

An  interesting  observation  by  ARL  personnel  is  that  trainees  often  have  incomplete  or 
inaccurate  information  about  the  consequences  of  their  actions  and  therefore  may  not  be  in  a 
position  to  self-evaluate  (Linda  Pierce,  personal  communication,  February,  1998).  However,  to 
the  contrary,  the  mini-AAR  process  can  counteract  these  inaccurate  perceptions  by  allowing 
various  points  of  view  to  surface  during  the  cell  mini-AAR  meetings.  If  one  cell  member  has 
limited  or  inaccurate  perceptions,  contrary  opinions  by  other  cell  members  can  serve  as  “mid¬ 
course”  corrections.  The  possibility  still  exists  that  all  cell  members  may  have  inaccurate 
information  about  the  consequences  of  their  performance.  Here  is  where  other  sources  of 
information  may  be  brought  to  bear:  expectations  vs.  actual  performance;  the  electronic  data 
stream  which  records  actual  events;  observations  and  judgments  of  0/Cs,  where  an  0/C  cadre  is 
available;  and  finally  the  perspectives  of  senior  leadership  and  training  managers.  The  challenge 
here  is  how  to  incorporate  these  other  sources  into  the  “brainstorming”  of  the  staff  cells  as  they 
go  through  their  mini- AARs. 
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Negative  Comments 


A  number  of  comments  question  the  need  for  and  the  value  of  the  mini-AAR 
methodology  under  research  investigation  by  ARI.'  The  comments  communicate  the  message 
that  we  shouldn’t  be  trying  to  fix  something  that’s  not  broken.  They  maintain  that  Army  policy 
and  practice  on  after  action  review  is  sufficient  and  requires  no  innovation.  The  comments 
further  suggest  that  command  leadership  has  the  tools  necessary  to  provide  training  feedback  and 
requires  no  additional  training  tools  or  innovations. 

“I  don’t  feel  we  need  a  supplement  to  the  guidelines  given  by  the  Army  on  how  to  conduct 
AARs.” 

“We  rely  on  officer  or  NCO  leadership  to  improve  performance.  We  know  our  strengths 
and  weaknesses  w/o  being  prompted  by  a  form.” 

“I  think  the  sections  are  good  at  finding  &  fixing  problems.  We  are  always  able  to  find 
problems  with  training.  It’s  never  hunky-dory.” 

“DivArty  could  easily  develop  one  on  their  own.  We  conduct  AARs  after  every  training 
event,  as  the  results  are  used  to  tailor  future  training.” 

“If  leaders  are  focused  then  important  areas  won’t  be  overlooked.” 

“I  think  that  a  leader-led  AAR  focused  on  the  METL  tasks  and  their  training  objectives  is 
sufficient.  Officers  and  NCOs  can  put  comments  on  paper  if  necessary,  but  a  long  form  is 
unnecessary  and  usually  ineffective.” 

“N/A  -  Didn’t  see  any  noticeable  improvement.” 

Some  thoughtful  comments  and  food  for  thought  are  provided  above.  There  may  be  a 
communication  problem  here,  a  misunderstanding  about  what  we’re  trying  to  achieve.  The 
misunderstanding  is  not  surprising,  since  we  were  not  able  to  actually  get  the  daily  mini-AARs 
implemented.  Trainees  did  not  experience  the  complete  feedback  process.  We  are  in  fact  trying 
to  provide  some  tools  to  support  cell  leader-led  AARs  focused  on  the  METL  tasks  and  training 
objectives.  Our  checklist  was  derived  from  METL  tasks  and  training  objectives.  We’re  also 
trying  to  develop  a  methodology  to  promote  timely  and  frequent  feedback  --  at  least  daily  if  not 
more  often,  as  suggested  in  comments  from  the  ARL/NAWCTSD  survey.  The  current  AARs 
would  be  sufficient  if,  in  fact,  they  were  done  in  a  systematic,  persistent,  and  timely  fashion.  At 
the  January  Fort  Hood  DivArty  exercises  there  were  no  section  AARs.  Whether  this  is  the  usual 
case,  or  was  a  special  case  because  of  the  ARL  research,  we  don’t  know.  We  do  know  from 
scientific  research  that  training  feedback  is  most  effective  when  given  close  to  where  the  training 
deficiencies  occur.  The  methodology  we  are  examining  is  designed  to  encourage  more  timely 
feedback  as  well  as  greater  participation  from  all  the  players. 
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Some  concern  was  expressed  about  the  value  of  filling  out  “self-assessment”  checklists. 
Again  I  think  we  didn’t  communicate  our  methodology  adequately.  We  need  to  explain  more 
effectively  that  we  are  developing  and  testing  an  assessment/feedback  system,  with  the  checklist 
as  a  component  of  the  system.  The  purpose  of  the  checklist  is  to  give  the  staff  cell  participants  a 
chance  to  document  their  ovra  perspectives  and  judgments  about  training  effectiveness  and 
problems,  and  then  use  the  checklists  to  contribute  when  the  cell  convenes  for  the  actual  mini- 
AAR.  The  checklist  is  designed  to  extract  maximal  benefit  from  each  participant’s  training 
experience.  We  agree  with  the  concerns  about  excessive  paper  work.  We’re  trying  to  keep  the 
paper  work  to  a  minimum.  A  research  goal  is  to  design  short,  simple  forms  that  get  the  job  done. 

Finally,  the  view  was  expressed  that  trainees  and  their  leaders  are  fully  capable  of 
identifying  and  fixing  training  problems  with  current  SOP  and  capabilities.  We  don’t  doubt  this 
in  the  least.  In  fact  those  capabilities  provide  a  key  premise  underlying  the  prototype  self- 
assessment/mini-AAR  methodology.  It  is  because  the  trainees  “know”  their  “strengths  and 
weaknesses”  that  the  self-assessment/mini-AAR  can  add  value  to  the  training.  The  methodology 
allows  each  member  of  the  cell  to  document  his  or  her  views  of  training  problems  and  helps 
bring  out  differences  of  opinion  for  constructive  discussion  when  the  cell  meets. 

Our  preliminary  data  show  frequent  same-task  disagreements  ranging  from  “needs 
improvement”  to  “commendable.”  So,  not  everyone  in  the  cell  will  see  strengths  and  weaknesses 
in  the  same  way.  Psychological  theory  indicates  that  well  qualified  personnel  observing  the 
same  events  may  focus  on  different  aspects  of  those  events  and  arrive  at  different  conclusions. 

The  value  of  our  methodology  is  that  it  encourages  discussion  to  discover  and  reconcile  the 
different  perceptions  and  cognitive  conclusions.  It  is  a  mechanism  for  approaching  ground  truth 
and  providing  feedback  based  on  that  truth. 

Constructive  Critiques-General 

Though  few  in  number,  a  set  of  comments  surfaced  some  interesting  methodological 
problems  that  we  need  to  attend  to:  (1)  carefully  tailoring  the  self  assessment  checklists  to  the 
needs  of  specific  staff  cells;  (2)  considering  the  roll  of  other  information  sources  such  as  the 
major  sequence  event  list  (MSEL),  and  information  from  0/Cs;  (3)  AAR  facilities,  i.e.,  the 
logistics  of  implementing  AARs;  and  (4)  providing  adequate  training  in  the  use  of  the 
methodology.  These  issues  were  suggested  by  the  following  comments: 

“The  self-assessments  should  be  tailored  more  towards  battle  operations  than  the  orders 
process.” 

“It  dealt  more  with  planning  than  execution.” 

“Must  be  aligned  with  duty  position.” 

“There  was  no  section  AAR  conducted.  If  each  section  was  given  a  copy  of  the  MEL  which 
affected  their  section,  they  could  have  a  good  AAR.” 
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“Use  desks,  not  chairs,  separate  officers  and  enlisted.  Give  the  soldiers  an  accurate 
understanding  of  what  they  are  to  do.” 

The  first  two  comments  above  are  from  the  Fire  Support  Element  (FSE)  and  merit 
special  discussion.  The  comments  show  a  concern  about  tasks  that  FSE  persoimel  didn't  think 
they  were  supposed  to  do.  The  Fire  Support  Element  (FSE)  self-assessment  tool  was  designed 
for  both  the  FSE  at  the  DMAIN  location  (involved  with  planning/orders  preparation)  and  the 
FSE  at  the  DTAC  location  (involved  with  current  battle  execution).  Since  the  exercise  at  Fort 
Hood  played  the  FSE  only  at  the  DTAC,  the  comments  referenced  above  are  understandable. 

For  future  exercises,  the  FSE  self-assessment  tool  could  be  written  in  two  versions:  one  for  FSE 
at  DMAIN  and  one  for  FSE  at  DTAC.  The  broader  lesson  learned  is  that  resources  must  be 
allocated  and  arrangements  made  to  staff  the  self-assessment  materials  with  the  types  of 
personnel  who  will  be  asked  to  use  them. 

The  comment  referencing  the  MEL  (Major  Event  List)  raises  an  interesting  question. 
Would  it  be  useful  to  work  the  MEL  into  the  mini-AAR/self  assessment  process  and  how  would 
it  be  best  to  do  so?  Our  self-assessment  checklists  are  task  and  training  objectives  oriented, 
rather  than  event-sequenced  like  the  MEL.  How  can  we  combine  the  two?  Maybe  the  cell  can 
begin  with  the  larger  task  judgments  and  then  go  to  the  MEL  to  get  specific  examples  of 
problems.  Is  there  some  way  to  “job-aid”  this  linkage?  ARL/NAWCTSD  tried  to  get  a  cross¬ 
walk  between  the  MEL  (i.e.,  TRACE)  and  more  generic  judgments  (TRACT)  and  didn’t  seem  to 
have  much  luck  doing  so.  But  it  might  be  easier  to  link  MEL  events  with  the  process  and 
product  statements  in  the  self-assessment  forms. 

The  last  comment  above  raises  three  issues:  (1)  logistics  of  administering  the  self- 
assessment/min-AAR;  (2)  composition  of  the  mini-AAR  group;  and  (3)  orientation  and  training 
in  the  purposes  and  procedures  of  the  self-assessment/mini-AAR.  We  need  to  try  to  arrange 
things  more  comfortably  and  make  it  easier  for  the  troops  to  carry  out  the  various  parts  of  the 
self-assessment/mini-AAR  process.  Desks  may  not  be  feasible,  especially  in  field  environments, 
but  maybe  a  table  can  be  set  up,  or  at  the  very  least,  clip  boards  provided  as  minimal  writing 
surfaces. 

Regarding  the  separation  of  officers  and  enlisted  personnel,  arguments  exist  on  both 
sides.  Enlisted  personnel  may  be  intimidated  by  officers.  On  the  other  hand,  if  they  are 
separated,  each  segment  will  not  have  the  benefit  of  the  other  segments  viewpoints.  The  value  of 
diversity  of  perspectives  and  ideas  is  a  key  premise  of  the  self-assessment/mini-AAR  process. 

The  final  clause  in  the  last  comment  is  especially  important.  In  fixture  tests  and 
applications,  significant  time  and  effort  should  be  expended  to  explain  the  self-assessment/mini- 
AAR  methodology.  A  cursory,  five-minute  overview  isn’t  going  to  get  the  job  done.  This  is  a 
bit  tricky.  We  don’t  want  to  burden  the  players  with  additional  training  requirements,  but  we 
have  to  insure  that  “the  soldiers  [have]  an  accurate  understanding  of  what  they  are  to  do.” 
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Constructive  Comments  on  the  Self-Assessment  Package  —  Instructions  and  Self-Assessment 
Checklist 


This  package  included  an  instruction  sheet,  a  self-assessment  checklist,  and  several  job 
aids:  a  flow  chart  showing  inputs  and  outputs  from  each  cell  to  every  other  cell  and  a  table 
describing  inputs,  related  processes,  and  outputs  for  each  staff  cell.  Survey  responses  and 
comments  indicated  that  the  instruction  sheet  and  the  self-assessment  checklist  were  positively 
received.  One  respondent,  however,  did  indicate  that  he  was  unaware  of  an  instruction  sheet. 
His  comment  reinforces  the  need  to  take  greater  care  and  expend  more  effort  in  explaining  the 
self-assessment  package. 

“Very  concise.”  [Instruction  Sheet] 

“What  instruction  sheet?” 

“They  [Self-Assessment  Checklist]  were  good  reminders.” 

“I  got  an  extra  one  [Self-Assessment  Checklist].” 

Constructive  Comments  on  the  Self-Assessment  Package  —  Job  Aids 

The  comments  below  indicate  a  range  of  opinions  on  the  usefulness  of  the  job  aids.  The 
tables  and  flow  charts  we  provided  may  be  more  useful  as  training  materials,  than  as  AAR 
guides.  But  without  having  observed  mini-AARs,  we  don’t  know  how  they  might  have  been 
used  during  the  discussions.  Cross-references  from  the  checklist  to  the  job  aids  may  increase  the 
usefulness  of  the  job  aids  during  the  mini -AAR. 

“Couldn’t  see  relevance  of  flow  chart  to  AAR.” 

“They  [flow  charts]  provided  good  details.” 

“My  section  is  very  talkative.” 

“They  [table  of  processes]  served  as  a  useful  or  useful  guides.” 

“I  know  how  to  do  an  AAR.” 
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Assessment  Checklist  Data 


Table  2  summarizes  responses  filled  out  on  assessment  checklists  by  members  of  three 
staff  cells,  on  the  first  two  days  of  a  three-day  Division  Artillery  Staff  Training  exercise.  A 
different  shift  of  personnel  participated  each  day,  but  a  few  people  participated  on  both  days. 


Table  2.  Percent  of  “Needs  Improvement”  Responses 


Day 

Staff  Element 

Percent  “Opportunity  for  Improvement” 

Product 

Process 

1 

Fire  Support  Element  (FSE)  [3] 

50%  (12/24) 

30%  (16/54) 

2 

Fire  Support  Element  [5] 

40%  (16/40) 

49%  (44/90) 

'  U  '2'  V' 

V 

1 

Fire  Control  Element  (FCE)  [4] 

4%  (1/24) 

16%  (12/77) 

2 

Fire  Control  Element  [3] 

11%  (2/18) 

28%  (15/54) 

■  ;s'. 

1 

Operations  (OPS)  [7] 

29%  (7/24) 

44%  (65/148) 

2 

Operations  [5] 

10%  (2/20) 

11%  (11/104) 

]  =  number  of  staff  cell  members  who  completed  the  checklist 


The  numbers  of  personnel  responding  to  the  checklists  are  given  in  square  brackets  under  “Staff 
Element.”  The  data  under  “Producf  ’  and  “Process”  reflect  percentages  of  “Opportunity  for 
Improvement”  responses.  The  baseline  for  the  percentages  includes  responses  to  all  the  checklist 
response  categories. 

Sensitivity  to  Training  Problems.  Note  in  Table  2  the  large  numbers  and  percentages  of 
processes  and  products  across  cells  and  days  judged  to  be  in  need  of  improvement.  These 
quantities  underscore  the  potential  value  of  the  players  as  data  sources  and  the  related  value  of 
the  self-assessment  scales  for  extracting  that  data.  That  is,  the  self-assessment  instrument 
appears  to  be  sensitive  to  perceived  training  deficiencies. 

Trend  Analysis  and  Comparisons.  It  is  not  meaningful  to  do  trend  analysis  or  comparison 
with  the  current  data  because  too  meiny  confounding  variables  were  operating  across  days.  For 
example,  different  shifts,  with  different  S3s  participated  on  Days  1  and  2,  though  some  personnel 
did  participate  on  both  days.  And  the  need  to  accommodate  a  test  of  a  new  simulation  facility 
added  its  own  complications,  not  normally  present  in  scheduled  training  events  for  division 
artillery  staff.  Finally,  the  reliability  and  validity  of  the  checklist  itself  are  unknown. 

But  the  results  illustrate  how  the  checklist  data  might  be  used  to  do  a  “temperature  check” 
of  training  progress  across  days  and  across  staff  cell  elements  under  more  controlled,  comparable 
conditions.  By  extension  these  same  analyses  might  be  used  for  comparing  alternative  training 
systems.  Imagine  then  that  the  checklist  data  were  obtained  under  circumstances  that  justified 
meaningful  comparisons  across  days  and  staff  cell  elements.  What  might  the  data  tell  us? 
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a.  Process  and  product  measures  appear  to  co-vary.  A  non-significant  chi-square  test  for 
product  vs.  process  across  days  and  staff  elements  supports  this. 

b.  The  OPS  cell  “judged”  itself  much  in  need  of  improvement  at  the  end  of  Day  1,  but 
“made  substantial  progress”  by  the  end  of  Day  2. 

c.  The  FSE  “experienced”  a  large  (the  largest)  proportion  of  perceived  training 
deficiencies  overall. 

d.  The  FCE  “reported”  the  smallest  amount  of  training  deficiency,  especially  for  the 
“Producf  ’  measure,  but  relatively  more  training  deficiencies  on  Day  2  than  on  Day  1 . 

Again,  it  needs  to  be  emphasized  that  the  above  comments  are  a  hypothetical  illustration 
of  how  self-assessment  data  might  be  summarized  and  used  to  get  a  “temperature  check”  on 
training  progress  and  needs.  The  comments  are  not  intended  to  characterize  the  staff  that 
participated  in  the  training  event. 
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•  DISCUSSION 


Evaluation  of  SAMAAR 

Value/Need.  The  survey  results  indicate  the  value  of  and  the  need  for  a  self-assessment 
approach  as  a  timely  and  feasible  way  to  support  training  feedback.  Reactions  to  the  checklist 
generally  were  favorable.  A  few  negative  reactions  were  aimed  at  use  of  the  checklist  in 
isolation  of  a  mini-AAR.  But  since  mini-AARs  were  not  carried  out,  the  purpose  of  the  checklist 
as  a  precursor  to  the  mini-AAR  was  perhaps  not  understood.  Some  participants  did  question  the 
need  to  fix  something  that  “wasn’t  broken.”  Their  reactions  suggested  that  a  clearer  explanation 
of  the  SAMAAR  approach  needs  to  be  included  in  future  trial  applications. 

Job  Aids.  Reactions  to  the  job  aids  included  in  the  self-assessment  package  were  mixed. 
Some  respondents  felt  that  the  flow  chart  and  input-process-output  table  were  useful;  others  did 
not.  The  role  of  such  aids  needs  further  thought  and  evaluation.  Perhaps  we  need  to  find  a  more 
explicit  way  to  link  these  job  aids  to  the  checklist  and  mini-AAR  and  then  explain  how  to  use 
them.  Cross-references  from  checklist  items  to  specific  sections  of  the  job  aids  might  be  useful. 
If  so  we  need  to  create  the  references  and  then  explain  their  use  during  the  orientation  phase  of 
the  test  bed  exercise.  If  we  were  to  distribute  similar  job  aids  in  follow-up  joint  training 
research,  we  should  do  so  as  part  of  the  pre-exercise  orientation  and  then  assess  their  utility  for 
both  exercise  preparation  and  as  an  aid  in  the  SAMAAR  process. 

Self-Assessment  Checklist  Data 

The  checklist  data  and  evaluation  results  indicated  that  the  checklist  was  easy  to  complete 
and  sensitive  to  training  problems.  Comments  and  checklist  responses  from  the  FSE  cell  did 
indicate  that  many  of  its  items  were  not  well  tailored  to  its  role  in  this  particular  exercise,  e.g., 
“The  self  assessments  should  be  tailored  more  towards  battle  operations  than  the  orders  process.” 
“It  dealt  more  with  planning  than  execution.”  But  these  results  only  underscore  the  importance 
of  very  careful  staffing  of  the  checklist  with  its  target  population  and  the  need  for  a  capability  to 
make  revisions  in  checklist  items  “at  the  last  minute.” 

In  the  Results  Section  we  presented  summaries  of  checklist  responses  across  trainees 
within  staff  cells  and  days.  The  purpose  for  presenting  these  summaries  was  to  illustrate  how 
such  data  might  be  used  to  do  trend  analyses  of  training  progress  across  days  and  comparisons  of 
training  progress  among  staff  cells  under  appropriately  standardized  conditions.  Appropriate 
conditions  for  trend  analysis  and  comparisons  did  not  exist  here.  There  were  too  many  artificial 
and  confounding  factors.  For  example,  the  exercise  was  a  test  bed  for  evaluating  a  stimulator- 
simulator,  normally  used  for  operational  training.  Its  unconventional  use  resulted  in  technical 
problems  that  delayed  or  interrupted  the  exercises  several  times.  But  under  appropriate 
conditions  the  checklist  data  might  be  useful  in  helping  to  monitor  training  progress.  They  might 
also  provide  a  mechanism  to  “red  tag”  recurring,  persistent  training  problems  across  rotations, 
problems  that  may  benefit  from  special  attention.  This  latter  use,  however,  may  be  problematic 
if  exercise  scenarios  are  not  standardized. 
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Implementation  Shortfalls 


The  trial  implementation  of  SAMAAR  fell  short  of  expectations.  The  shortfalls  were,  in 
part,  the  result  of  limited  control  of  scheduling  the  SAMAAR  events  and  administration  of 
research  materials.  For  example,  in  the  interest  of  “non-interference”  the  checklist  forms  were 
given  to  staff  personnel  for  distribution  to  and  collection  from  staff  cell  members.  A  number  of 
consequences  resulted.  Completed  checklist  forms  were  received  from  only  three  of  the  four 
staff  elements.  Heading  information,  e.g.,  date,  time,  cell,  duty  position  was  missing  on  many 
otherwise  completed  forms.  Only  eight  participants  filled  out  SAMAAR  evaluation  forms.  Most 
significant,  mini-AARs  that  were  anticipated  at  the  end  of  each  of  the  three  mission  days  did  not 
occur.  But  the  shortfalls  did  provide  constructive  “lessons-leamed”  for  follow  on  research. 

To  be  worth  the  effort,  future  SAMAAR  trial  applications  need  to  be  conducted  under  a 
direct,  pre-arranged  understanding  of  research  requirements  and  agreement  of  support  by  the  host 
unit.  Such  research  can  still  be  carried  out  on  a  relatively  non-interfering  basis.  However,  it’s 
essential  to  maintain  a  roster  of  participants  in  order  to  track  application  of  SAMAAR  and  other 
research  materials.  It  would  still  be  possible  to  keep  completed  forms  anonymous,  if  desired. 

Future  enhancements  of  the  SAMAAR  approach  could  include  assessment  of  cell 
performance  by  those  receiving  cell  outputs.  A  research  application  might  include  assessments 
by  observer/trainers  (0/Ts).  A  comparison  of  their  assessments  with  those  of  the  trainees  could 
provide  a  basis  for  validity  and  reliability  analyses.  Figure  1  outlines  these  enhancements  (Guy 
Siebold,  personal  communication,  February  1998).  In  addition  the  role  of  electronic  data  from 
digital  force  training  exercises  needs  to  be  examined  for  its  usefulness  in  assessing  the  training  of 
mission  planning  and  management. 


2.  Cell  as  a  group 
assesses  performance. 


1.  Cell  members  each 
assess  group  performance. 


Member 

Member 

Member 

Member 

Observer/ 1 
Trainer 


3.  Those  receiving 
cell  outputs  assess 
the  outputs. 


4.  Observer/Trainer  assesses 
group  performance  of  cell 
and  of  cell’s  mini-AAR. 


Figure  1 .  Future  enhancement  of  the  SAMAAR  approach. 
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CONCLUSIONS 


1 .  The  SAMAAR  approach  was  judged  to  be  useful  and  timely  for  providing  feedback  and 
surfacing  training  problems  that  otherwise  might  be  overlooked. 

2.  The  self-assessment  checklist  was  judged  to  be  clear  and  easy  to  complete. 

3.  The  checklist  data  under  standardized  conditions  can  provide  a  way  to  track  training  progress 
across  unit  elements  and  mission  days 

4.  The  value  and  appropriate  use  of  the  job  aids,  task  flow  charts  and  tables,  are  inconclusive. 
They  require  further  development  and  test. 
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APPENDIX  A 

TARGET  AUDIENCE  FOR  SELF-ASSESSMENT  PILOT  TEST 


OPERATIONS  (OPS) 

RANK 

Operations  Officer  (S-3) 

MAJ 

Counter  fire  Officer 

CPT 

Operations  Sgt 

SGM 

Chemical  Officer 

CPT 

Clerk/RTO 

SPC 

Reconnaissance  Survey  Officer 

CPT 

Surveyor 

SFC 

NOTES 


FIRE  CONTROL  ELEMENT  (FCE) 

RANK 

Fire  Control  Officer 

CPT 

Fire  Control  NCO 

SFC 

Computer  Operator 

SSG 

Tacfire  Equipment  Spec 

PFC 

Tacfire  Equipment  Spec 

PFC 

Tacfire  Equipment  Spec 

PFC 

TARGETING  ELEMENT  (TGTG) 

RANK 

Intel  Officer 

CPT 

Processing  Chief 

SSG 

Target  Processing  NCO 

SGT 

Target  Processing  Spec 

PFC 

Target  Processing  Spec 

SPC 

Intel  SGT  (NCOIC) 

MSG 

Intel  Analyst 

SSG 

Intel  Analyst 

SGT 

1 

Intel  Analyst 

PFC 

DTAC  FIRE  SUPPORT  ELEMENT 
(FSE) 

RANK 

Assistant  FS  Coordinator 

CPT 

Fire  Support  SGT 

SFC 

Fire  Support  NCO 

SGT 

Clerk 

SPC 

Fire  Support  Specialist 

SPC 
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APPENDIX  B 

COMPONENTS  OF  THE  SELF-ASSESSMENT  PACKAGE 


Package  Page 

Background  and  Purpose  of  Tools . B-2 

Operations  Section . B-OPS-1 

Fire  Control  Element . B-FCE-1 

Targeting  Section . B-TGT-1 

DTAC  Fire  Support  Element . B-FSE-1 
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Self-Assessment  Tools  for 
Division  Artillery  Staff  Sections 

1 .  Background.  U.S.  Army  Research  Institute  (ARI)  has  been  developing  a  concept  for 
measuring  performance  during  training  of  high-level  (Joint  Task  Force  and  above)  joint  staffs. 
For  staff  sections  at  these  levels,  it  is  difficult  to  obtain  outside  observer/ 

controllers  to  provide  meaningful  feedback  during  the  training  of  these  sections.  ARI’s  basic 
premise  is  that  staff  sections  possess  the  expertise  to  assess  their  own  performance,  and  that  if 
provided  the  proper  self-assessment  tools,  they  can  conduct  effective  mini-AARs  to  improve 
their  training.  This  concept  was  tested  during  a  major  CINCCENT  exercise  in  1997,  with  very 
favorable  results. 

2.  Purpose.  ARI  would  like  to  try  out  a  set  of  self-assessment  tools  developed  for  use  by  the 
key  staff  sections  involved  with  the  fire  support  operations  of  a  division  artillery.  This  is 
completely  voluntary  on  the  part  of  the  unit. 

a.  Enclosed  are  four  sets  of  tools  each  with  an  introduction  to  the  self-assessment  tools 
themselves,  instructions  for  their  use,  and  diagrams  of  inputs,  outputs,  and  processes.  There  is  a 
set  for  the  operations  section,  the  targeting  section,  the  fire  control  element,  and  the  fire  support 
element.  Many  of  the  functions  these  sections  perform  are  process  oriented,  as  opposed  to 
outcome  oriented.  To  perform  the  processes  involves,  to  a  large  degree,  the  performance  of 
military  “art”  based  on  collective  judgment.  This  is  something  difficult  to  measure  objectively. 

It  was,  therefore,  necessary  to  derive  mainly  subjective  performance  measurement  tools. 

b.  The  self-assessment  tools  are  for  the  use  of  the  senior  officer  in  charge  of  each  staff  section 
to  conduct  a  10-15  minute  mini-AAR  (after  action  review)  at  the  conclusion  of  each  iteration  or 
phase  of  the  section’s  fire  support  processes.  This  allows  him  to  give  and  receive  feedback  on 
how  well  the  process  was  performed  and  on  what  needs  to  be  fixed  or  improved  for  the  next 
iteration.  The  self-assessment  tools  are  in  the  form  of  questions  to  ask  oneself  and  the  section 
members.  Their  purpose  is  to: 


1)  Provide  a  reminder  and  a  format  to  conduct  on-going,  semi-structured  reviews  of 
what’s  being  done,  for  what  purpose,  and  how  well  it’s  being  done. 

2)  Provide  a  record  of  performance  progress  throughout  the  duration  of  a  training 
exercise  or  during  actual  combat. 

3)  Provide  a  record  of  lessons  learned  and  critical  events  for  inclusion  into  the  overall 
DivArty  AAR. 

4)  Identify  tasks  which  require  additional  training  or  need  added  emphasis  in  future 
iterations  of  the  process  being  performed  eind  in  future  training  exercises. 

3.  At  the  conclusion  of  the  training  exercise,  ARI  would  like  to  receive  feedback  from  the  staff 
sections  on  the  self-assessment  process  in  general  and  on  the  individual  tools  in  particular. 
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Training  Objective  Self-Assessment  Tools  for 
Division  Artillery  Operations  Section 
Mini-AARs 


1 .  Training  Objective.  Train  the  Operations  Section  in  planning  and  controlling  the 
employment  of  FA  unit  operations.  Major  tasks  involved  are: 

a.  Prepare  the  FA  support  plan. 

b.  Direct  and  coordinate  FA  unit  operations. 

c.  Plan  employment  of  FA  assets. 

d.  Collect  and  manage  FA  tactical  information. 

2.  Self-Assessment  and  Feedback  (Background/Purpose). 

a.  One  of  the  major  challenges  to  assessing  training  performance  and  giving  meaningful 
feedback  is  finding  the  right  people  with  the  right  expertise  to  conduct  the  assessment.  At  lower 
echelons,  those  personnel  can  be  found  in  like  units  or  in  higher  echelon  units  and  can  be  tasked 
to  perform  assessment  and  feedback  for  other  units’  training.  But  for  a  division,  it  is  often  times 
impractical  to  task  like  or  higher  units  outside  of  the  division  for  personnel.  Additionally,  during 
actual  combat  or  for  training  at  times  other  than  during  a  major  exercise,  expert 
observers/controllers/trainers  will  not  be  present  to  provide  the  necessary  feedback  for 
improvement.  So  where  does  one  find  the  required  expertise?  It  can  be  found  within  the  unit’s 
own  staff  persormel.  Therefore,  the  training  performance  measures  and  feedback  methods 
contained  in  this  guide  were  designed  for  self-assessment  by  personnel  in  those  division  artillery 
staff  elements  involved  with  fire  support  operations. 

b.  Many  of  the  tasks  performed  by  DivArty  staff  elements  in  fire  support  operations  are 
process  oriented,  as  opposed  to  outcome  oriented  (see  page  OPS-7).  To  perform  the  processes 
involves,  to  a  large  degree,  the  performance  of  military  “art”  based  on  collective  judgment.  This 
is  something  difficult  to  measure  objectively.  It  was,  therefore,  necessary  to  derive  mainly 
subjective  performance  measurement  tools. 

c.  The  Self-Assessment  Tools  are  to  be  used  by  the  senior  officer  in  charge  of  each  shift  of 
the  operations  section  to  conduct  a  10-15  minute  mini- AAR  (after  action  review)  for  his  shift  at 
the  conclusion  of  each  iteration  or  phase  of  its  fire  support  processes.  This  allows  him  to  give 
and  receive  feedback  on  how  well  the  process  was  performed  and  on  what  needs  to  be  fixed  or 
improved  for  the  next  iteration.  The  Self-Assessment  Tools: 

1)  Provide  a  reminder  and  a  format  to  conduct  on-going,  semi-structured  reviews  of 
what’s  being  done,  for  what  purpose,  and  how  well  it’s  being  done. 

2)  Provide  a  record  of  performance  progress  throughout  the  duration  of  a  training 
exercise  or  during  actual  combat. 
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3)  Provide  a  record  of  lessons  learned  for  inclusion  into  the  overall  DivArty  AAR. 

4)  Identify  tasks  which  require  additional  training  or  need  added  emphasis  in  future 
iterations  of  the  process  being  performed  and  in  future  training  exercises. 

3.  Instructions.  The  attached  Self-Assessment  Tool  (pages  OPS-3  and  OPS-4)  for  the 
operations  section  (OPS)  is  in  the  form  of  questions  for  the  senior  officer  in  charge  of  each  OPS 
shift  to  ask  himself  and  the  members  of  his  shift  while  conducting  a  mini-AAR  prior  to  the 
conclusion  of  each  shift  or  after  a  major  process  has  been  accomplished.  Pages  OPS-5  and  OPS- 
6  are  provided  as  a  ready  reference  and  show  input/output  flow  for  the  DivArty  as  a  whole  and 
for  the  OPS  section  in  particular. 

1)  The  date/time  and  person  conducting  the  mini-AAR  should  be  filled  in,  so  that  a 
record  of  progress  can  be  maintained,  and  any  future  questions  can  be  directed  to  the 
appropriate  person. 

2)  Any  of  the  “Products/Outputs”  which  were  developed  should  be  rated  as  to  how 
effective  they  are. 

3)  Each  of  the  questions  under  “Assessment  of  Processes”  should  be  discussed  and 
rated.  Any  examples  of  significant  or  critical  events  arising  from  these  discussions 
should  be  written  in  the  “Comments”  section  on  the  back  or  attached  on  separate  sheets. 

4)  Lessons  learned,  fixes  to  problems,  or  other  relevant  information  discovered  during 
the  mini-AAR  should  be  addressed  with  the  on-coming  shift. 
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DivArty  OPS  Section 


Assessment  of  Products/Outputs 


Field  Artillery  Support  Plan 


DivArty  commander's  guidance 


Movement  orders 


Mission  changes 


Assessment  of  Processes 


How  well  were  FA  assets  allocated  and  prioritized? 


Were  changes  in  allocation  projected  overtime  lAW  div  OPORD? 


Were  instructions  for  timely  detection/attack  of  HPTs  given? 


Were  requirements  for  positioning  of  all  FA  assets  determined? 


Were  make-up  of  basic  loads  &  the  RSR  determined? 


Were  locations  of  ASPs  and  ATPs  disseminated? 


Were  cmdr’s  attack  guidance  &  tgt  select  stnd  determined? 


Was  effective  control  of  fire  control  operations  maintained? 


How  well  was  the  coordination  of  target  acquisition  directed? 


How  well  were  the  planning  and  execution  of  survey  conducted? 


How  well  was  the  movement  of  units  managed? 


Was  DivArty  cmdr  kept  informed  and  guidance  sought  from  him? 


How  effective  was  commo  w/higher,  lower,  and  adjacent  units? 
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DivArty  OPS  Section 
(Continued) 


Assessment  of  Processes  (Continued) 


How  well  were  appropriate  inputs  and  documents  used? 


Were  appropriate  personnel  (external/internal)  consulted? 


How  well  did  key  personnel  provide  relevant  input? 


Were  doctrinal  principles  and  cmdr's  philosophies  adhered  to? 


Were  matters  sufficiently  discussed?  Alternatives  addressed? 


Was  each  output  consistent  with  cmdr's  guidance/intent? 


Were  outputs  produced  and  distributed  in  a  timely  manner? 


Was  a  “mini-AAR”  conducted  w/in  the  S-3  Ops  after  each  activity? 


Comments  (fixes/lessons  learned/critical  incidents/examples): 
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Input/Output  Flow  Among  DivArty  Elements 


B-OPS-5 


Input/Output  Flow  Among  DivArty  Elements 

Operations  (OPS) 


B-OPS-6 


Inputs:  Process:  Prepare  the  Field  Artillery  Support  Plan  Output/Product: 
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element  with  special  consideration  to  counterfire  targets.  -  Fire  missions 

5.  Plan,  direct,  coordinate,  and  control  the  fires  of  organic,  attached,  and  -  Movement  orders 

reinforcing  field  artillery  units.  -  Mission  changes 

6.  Coordinate  and  direct  target  acquisition  and  survey  operations. 


Training  Objective  Self-Assessment  Tools  for 
Division  Artillery  Fire  Control  Element 
Mini-AARs 


1 .  Training  Objective.  Train  the  FCE  Section  in  providing  timely,  accurate,  and 
effective  technical  fire  solutions  and  tactical  fire  control  of  FA  fires.  Major  tasks 
involved  are: 

a.  Control  and  coordinate  FA  fire  missions. 

b.  Prepare  schedules  of  fire. 

c.  Control  target  attack. 

d.  Provide  counterfire. 

e.  Perform  and  provide  target  value  analysis. 

2.  Self-Assessment  and  Feedback  (Background/Purpose). 

a.  One  of  the  major  challenges  to  assessing  training  performance  and  giving 
meaningful  feedback  is  finding  the  right  people  with  the  right  expertise  to  conduct  the 
assessment.  At  lower  echelons,  those  personnel  can  be  found  in  like  units  or  in  higher 
echelon  units  and  can  be  tasked  to  perform  assessment  and  feedback  for  other  units’ 
training.  But  for  a  division,  it  is  often  times  impractical  to  task  like  or  higher  units 
outside  of  the  division  for  personnel.  Additionally,  during  actual  combat  or  for  training 
at  times  other  than  during  a  major  joint  exercise,  expert  observers/controllers/trainers  will 
not  be  present  to  provide  the  necessary  feedback  for  improvement.  So  where  does  one 
find  the  required  expertise?  It  can  be  found  within  the  unit’s  own  staff  personnel. 
Therefore,  the  training  performance  measures  and  feedback  methods  contained  in  this 
guide  were  designed  for  self-assessment  by  personnel  in  those  division  artillery  staff 
elements  involved  with  fire  support  operations. 

b.  Many  of  the  tasks  performed  by  DivArty  staff  elements  in  fire  support  operations 
are  process  oriented,  as  opposed  to  outcome  oriented  (see  page  FCE-7).  To  perform  the 
processes  involves,  to  a  large  degree,  the  performance  of  military  “art”  based  on 
collective  judgment.  This  is  something  difficult  to  measure  objectively.  It  was,  therefore, 
necessary  to  derive  mainly  subjective  performance  measurement  tools. 

c.  The  Self-Assessment  Tools  are  to  be  used  by  the  senior  officer  in  charge  of  each 
FCE  shift  to  conduct  a  10-15  minute  mini-AAR  (after  action  review)  for  his  shift  at  the 
conclusion  of  each  iteration  or  phase  of  its  fire  support  processes.  This  allows  him  to 
give  and  receive  feedback  on  how  well  the  process  was  performed  and  on  what  needs  to 
be  fixed  or  improved  for  the  next  iteration.  The  Self-Assessment  Tools: 

1)  Provide  a  reminder  and  a  format  to  conduct  on-going,  semi-structured  reviews 
of  what’s  being  done,  for  what  purpose,  and  how  well  it’s  being  done. 
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2)  Provide  a  record  of  performance  progress  throughout  the  duration  of  a  training 
exercise  or  during  actual  combat. 

3)  Provide  a  record  of  lessons  learned  for  inclusion  into  the  overall  DivArty 
AAR. 

4)  Identify  tasks  which  require  additional  training  or  need  added  emphasis  in 
future  iterations  of  the  process  being  performed  and  in  future  training  exercises. 

3.  Instructions.  The  attached  Self-Assessment  Tool  (pages  FCE-3  and  FCE-4)  for  the 
fire  control  element  (FCE)  is  in  the  form  of  questions  for  the  senior  officer  in  charge  of 
each  FCE  shift  to  ask  himself  and  the  members  of  his  shift  while  conducting  a  mini- AAR 
prior  to  the  conclusion  of  each  shift  or  after  a  major  process  has  been  accomplished. 

Pages  FCE-5  and  FCE-6  are  provided  as  a  ready  reference  and  show  input/output  flow  for 
the  DivArty  as  a  whole  and  for  the  FCE  section  in  particular. 

1)  The  date/time  and  person  conducting  the  mini-AAR  should  be  filled  in,  so  that 
a  record  of  progress  can  be  maintained,  and  any  future  questions  can  be  directed 
to  the  appropriate  person. 

2)  Any  of  the  “Products/Outputs”  which  were  developed  should  be  rated  as  to 
how  effective  they  are. 

3)  Each  of  the  questions  under  “Assessment  of  Processes”  should  be  discussed 
and  rated.  Any  examples  of  significant  or  critical  events  arising  from  these 
discussions  should  be  ■written  in  the  “Comments”  section  on  the  back  or  attached 
on  separate  sheets. 

4)  Lessons  learned,  fixes  to  problems,  or  other  relevant  information  discovered 
during  the  mini-AAR  should  be  addressed  with  the  on-coming  shift. 
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DivArty  Fire  Control  Element  (FCE) 


Assessment  of  Processes 


Were  fire  msns,  tgt  lists,  &  schedules  sent  to  appropriate  units? 


Were  tgts  which  could  not  be  engaged  passed  to  FSE? 


Were  fires  coordinated  with  adjacent  units? 


Were  ammo  files  updated  upon  completion  of  fire  msns? 


Were  clearances  requested  from  FSE  when  necessary? 


Were  JAAT  fire  support  msns  coordinated  w/FSE? 


Was  fire  unit  status  monitored,  updated,  and  disseminated? 


Was  FLOT  continuously  updated? 


Were  fire  msn  tactical  solutions  disseminated  to  appropriate  units? 


Were  counterfire  templates  input  to  counterfire  schedules? 


Were  requests  for  EW,  CAS,  and  NGF  support  passed  to  FSE? 


Were  reinforcing/attached  FA  brigades  kept  informed/updated? 


DivArty  Fire  Control  Element  (FCE) 
(Continued) 
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Assessment  of  Processes  (Continued) 


How  well  were  appropriate  inputs  and  documents  used? 


Were  appropriate  personnel  (external/internal)  consulted? 


How  well  did  key  personnei  provide  relevant  input? 


Were  doctrinal  principles  and  cmdr's  philosophies  adhered  to? 


Were  matters  sufficiently  discussed?  Alternatives  addressed? 


Was  each  output  consistent  with  cmdr’s  guidance/intent? 


Were  outputs  produced  and  distributed  in  a  timely  manner? 


Was  a  “mini-AAR”  conducted  w/in  the  FCE  after  each  activity? 


Comments  (fixes/lessons  learned/critical  incidents/examples): 


Input/Output  Flo\V  Among  DivArty  Elements 


Input/Output  Flow  Among  DivArty  Elements 
Fire  Control  Element  (FCE) 


FCE-6 


DivArtv  Fire  Control  Element  (FCE) 
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Training  Objective  Self-Assessment  Tools  for 
Division  Artillery  Targeting  Section 
Mini-AARs 


1 .  Training  Objective.  Train  the  Targeting  Section  in  how  to  accomplish  the  decide- 
detect-deliver-assess  process.  Major  tasks  involved  are; 

a.  Conduct  intelligence  preparation  of  the  battlefield. 

b.  Develop  targets  and  potential  targets. 

c.  Predict  and  produce  targets. 

d.  Coordinate  target  acquisition  and  counterfire. 

2.  Self-Assessment  and  Feedback  (Background/Purpose). 

a.  One  of  the  major  challenges  to  assessing  training  performance  and  giving 
meaningful  feedback  is  finding  the  right  people  with  the  right  expertise  to  conduct  the 
assessment.  At  lower  echelons,  those  personnel  can  be  found  in  like  units  or  in  higher 
echelon  units  and  can  be  tasked  to  perform  assessment  and  feedback  for  other  units’ 
training.  But  for  a  division,  it  is  often  times  impractical  to  task  like  or  higher  units 
outside  of  the  division  for  personnel.  Additionally,  during  actual  combat  or  for  training 
at  times  other  than  during  a  major  joint  exercise,  expert  observers/controllers/trainers  will 
not  be  present  to  provide  the  necessary  feedback  for  improvement.  So  where  does  one 
find  the  required  expertise?  It  can  be  found  within  the  unit’s  own  staff  personnel. 
Therefore,  the  training  performance  measures  and  feedback  methods  contained  in  this 
guide  were  designed  for  self-assessment  by  persormel  in  those  division  artillery  staff 
elements  involved  with  fire  support  operations. 

b.  Many  of  the  tasks  performed  by  DivArty  staff  elements  in  fire  support  operations 
are  process  oriented,  as  opposed  to  outcome  oriented  (see  page  TGT-7).  To  perform  the 
processes  involves,  to  a  large  degree,  the  performance  of  military  “art”  based  on 
collective  judgment.  This  is  something  difficult  to  measure  objectively.  It  was,  therefore, 
necessary  to  derive  mainly  subjective  performance  measurement  tools. 

c.  The  Self-Assessment  Tools  are  to  be  used  by  the  senior  officer  in  charge  of  each 
shift  of  the  targeting  section  to  conduct  a  10-15  minute  mini- AAR  (after  action  review) 
for  his  shift  at  the  conclusion  of  each  iteration  or  phase  of  its  fire  support  processes.  This 
allows  him  to  give  and  receive  feedback  on  how  well  the  process  was  performed  and  on 
what  needs  to  be  fixed  or  improved  for  the  next  iteration.  The  Self-Assessment  Tools: 

1)  Provide  a  reminder  and  a  format  to  conduct  on-going,  semi-structured  reviews 
of  what’s  being  done,  for  what  purpose,  and  how  well  it’s  being  done. 

2)  Provide  a  record  of  performance  progress  throughout  the  duration  of  a  training 
exercise  or  during  actual  combat. 
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3)  Provide  a  record  of  lessons  learned  for  inclusion  into  the  overall  DivArty 
AAR. 

4)  Identify  tasks  which  require  additional  training  or  need  added  emphasis  in 
future  iterations  of  the  process  being  performed  and  in  future  training  exercises. 

3.  Instructions.  The  attached  Self-Assessment  Tool  (pages  TGT-3  and  TGT-4)  for  the 
targeting  section  (TGT)  is  in  the  form  of  questions  for  the  senior  officer  in  charge  of  each 
TGT  shift  to  ask  himself  and  the  members  of  his  shift  while  conducting  a  mini- AAR  prior 
to  the  conclusion  of  each  shift  or  after  a  major  process  has  been  accomplished.  Pages 
TGT-5  and  TGT-6  are  provided  as  a  ready  reference  and  show  input/output  flow  for  the 
DivArty  as  a  whole  and  for  the  TGT  section  in  particular. 

1)  The  date/time  and  person  conducting  the  mini- AAR  should  be  filled  in,  so  that 
a  record  of  progress  can  be  maintained,  and  any  future  questions  can  be  directed 
to  the  appropriate  person. 

2)  Any  of  the  “Products/Outputs”  which  were  developed  should  be  rated  as  to 
how  effective  they  are. 

3)  Each  of  the  questions  under  “Assessment  of  Processes”  should  be  discussed 
cind  rated.  Any  examples  of  significant  or  critical  events  arising  from  these 
discussions  should  be  written  in  the  “Comments”  section  on  the  back  or  attached 
on  separate  sheets. 

4)  Lessons  learned,  fixes  to  problems,  or  other  relevant  information  discovered 
during  the  mini-AAR  should  be  addressed  with  the  on-coming  shift. 
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DivArty  Targeting  Section 


Assessment  of  Products/Outputs 


Counterfire  templates 

Intel  data  base 

High  Value  Targets  (HVTs) 

Critical  nodes 

NAIs  &  TAIs 

Intel  estimates/annex 

Target  acquisition  missions 

Radar  Deployment  Orders  (RDO) 

Requests  for  additional  Intel  collection  assets 

Assessment  of  Processes 

How  well  were  critical  nodes,  NAIs,  TAIs,  &  HVTs  identified? 

Were  G2  &  FSE  consulted  during  FA  IPB  process? 

How  effective  was  development  of  targets  and  potential  targets? 

Were  positioning  and  relocation  of  TA  assets  recommended  to  S3? 

Did  target  value  analysis  determine  high  value  targets  (HVTs)? 

Were  high  payoff  targets  (HPTs)  recommended  to  S3  and  FSE? 

Were  radar  deployment  orders  effectively  prepared? 

Were  counterfire  templates  prepared  and  passed  to  FCE? 

Was  BDA  requested  and  BDA  analysis  conducted  for  enemy  FA? 

■ 

■ 

■ 
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DivArty  Targeting  Section 
(Continued) 


I_L 

Assessment  of  Processes  (Continued) 


How  well  were  appropriate  inputs  and  documents  used? 


Were  appropriate  personnel  (external/internal)  consulted? 


How  well  did  key  personnel  provide  relevant  input? 


Were  doctrinal  principles  and  cmdr's  philosophies  adhered  to? 


Were  matters  sufficiently  discussed?  Alternatives  addressed? 


Was  each  output  consistent  with  cmdr's  guidance/intent? 


Were  outputs  produced  and  distributed  in  a  timely  manner? 


Was  a  “mini-AAR”  conducted  w/in  S2  Tgt'ing  after  each  activity? 


Comments  (fixes/lessons  learned/critical  incidents/examples); 


Input/Output  Flow  Among  DivArty  Elements 


Input/Output  Flow  Among  DivArty  Elements 

Targeting  (TGTG) 
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Inputs:  Process:  Conduct  Intelligence  Preparation  of  the  Battlefield  (IPB)  Output/Product: 
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Training  Objective  Self-Assessment  Tools  for 
Division  Fire  Support  Element 
Mini-AARs 


1 .  Training  Objective.  Train  the  FSE  Section  in  the  planning,  coordination, 
integration,  and  synchronization  of  fire  support  operations.  Major  tasks  involved  are: 

a.  Conduct  fire  support  coordination  in  support  of  ground  operations. 

b.  Synchronize  fire  support  operations. 

c.  Conduct  fire  support  planning. 

d.  Coordinate  target  attack. 

e.  Process  target  attack. 

2.  Self-Assessment  and  Feedback  (Background/Purpose). 

a.  One  of  the  major  challenges  to  assessing  training  performance  and  giving 
meaningful  feedback  is  finding  the  right  people  with  the  right  expertise  to  conduct  the 
assessment.  At  lower  echelons,  those  personnel  can  be  found  in  like  units  or  in  higher 
echelon  units  and  can  be  tasked  to  perform  assessment  and  feedback  for  other  units’ 
training.  But  for  a  division,  it  is  often  times  impractical  to  task  like  or  higher  units 
outside  of  the  division  for  personnel.  Additionally,  during  actual  combat  or  for  training 
at  times  other  than  during  a  major  joint  exercise,  expert  observers/controllers/trainers  will 
not  be  present  to  provide  the  necessary  feedback  for  improvement.  So  where  does  one 
find  the  required  expertise?  It  can  be  found  within  the  unit’s  own  staff  personnel. 
Therefore,  the  training  performance  measures  and  feedback  methods  contained  in  this 
guide  were  designed  for  self-assessment  by  personnel  in  those  division  artillery  staff 
elements  involved  with  fire  support  operations. 

b.  Many  of  the  tasks  performed  by  DivArty  staff  elements  in  fire  support  operations 
are  process  oriented,  as  opposed  to  outcome  oriented  (see  page  FSE-7).  To  perform  the 
processes  involves,  to  a  large  degree,  the  performance  of  military  “art”  based  on 
collective  judgment.  This  is  something  difficult  to  measure  objectively.  It  was,  therefore, 
necessary  to  derive  mainly  subjective  performance  measurement  tools. 

c.  The  Self-Assessment  Tools  are  to  be  used  by  the  senior  officer  in  charge  of  each 
FSE  shift  at  both  the  DMAIN  and  DTAC  to  conduct  a  10-15  minute  mini-AAR  (after 
action  review)  for  his  shift  at  the  conclusion  of  each  iteration  or  phase  of  its  fire  support 
processes.  This  allows  him  to  give  and  receive  feedback  on  how  well  the  process  was 
performed  and  on  what  needs  to  be  fixed  or  improved  for  the  next  iteration.  The  Self- 
Assessment  Tools: 

1)  Provide  a  reminder  and  a  format  to  conduet  on-going,  semi-structured  reviews 
of  what’s  being  done,  for  what  purpose,  and  how  well  it’s  being  done. 
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2)  Provide  a  record  of  performance  progress  throughout  the  duration  of  a  training 
exercise  or  during  actual  combat. 

3)  Provide  a  record  of  lessons  learned  for  inclusion  into  the  overall  DivArty 
AAR. 

4)  Identity  tasks  which  require  additional  training  or  need  added  emphasis  in 
future  iterations  of  the  process  being  performed  and  in  future  training  exercises. 

3.  Instructions.  The  attached  Self-Assessment  Tool  (pages  FSE-3  and  FSE-4)  for  the 
fire  support  element  (FSE)  is  in  the  form  of  questions  for  the  senior  officer  in  charge  of 
each  FSE  shift  at  both  the  DMAIN  and  DTAC  to  ask  himself  and  the  members  of  his 
shift  while  conducting  a  mini- AAR  prior  to  the  conclusion  of  each  shift  or  after  a  major 
process  has  been  accomplished.  Pages  FSE-5  and  FSE-6  are  provided  as  a  ready 
reference  and  show  input/output  flow  for  the  DivArty  as  a  whole  and  for  the  FSE  section 
in  particular. 

1)  The  date/time  and  person  conducting  the  mini -AAR  should  be  filled  in,  so  that 
a  record  of  progress  can  be  maintained,  and  any  future  questions  can  be  directed 
to  the  appropriate  person. 

2)  Any  of  the  “Products/Outputs”  which  were  developed  should  be  rated  as  to 
how  effective  they  are. 

3)  Each  of  the  questions  under  “Assessment  of  Processes”  should  be  discussed 
and  rated.  Any  examples  of  significant  or  critical  events  arising  from  these 
discussions  should  be  written  in  the  “Comments”  section  on  the  back  or  attached 
on  separate  sheets. 

4)  Lessons  learned,  fixes  to  problems,  or  other  relevant  information  discovered 
during  the  mini- AAR  should  be  addressed  with  the  on-coming  shift. 
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DivArty  Fire  Support  Element  (FSE) 


Assessment  of  Products/Outputs 


Was  FS  planning  conducted  simultaneously  w/maneuver  planning? 

■ 

■ 

■ 

Were  all  FS  portions  of  Corps  OPORD  considered? 

Did  ALO  provide  necessary  sortie  information? 

Were  G-2's  IPB  products  considered  and  passed  to  DivArty? 

Was  DivArty  Cmdr's  guidance  solicited?  ...used? 

Did  FS  coordination  support  cmdr's  intent/concept  of  operations? 

Was  FS  planning  and  execution  synchronized  w/other  forces? 

Were  attacks  on  tgt’s  coordinated  and  processed  effectively? 

Was  coordination  with  higher  &  adjacent  units  maintained? 

Was  execution  of  FS  matrix  monitored  and  refined? 

■ 

■ 

■ 

■ 

DivArty  Fire  Support  Element  (FSE) 
(Continued) 


/  ^  /  •S  /  J . 

/  ^  ^  /  -i 

i  M  i  i 

t  i  9  Si 


Assessment  of  Processes  (Continued) 

How  well  were  appropriate  inputs  and  documents  used? 

■■■■ 

Were  appropriate  personnel  (external/internal)  consulted? 

■■■■ 

How  well  did  key  personnel  provide  relevant  input? 

■■■■ 

Were  doctrinal  principles  and  cmdr's  philosophies  adhered  to? 

Were  matters  sufficiently  discussed?  Alternatives  addressed? 

■■■■ 

Was  each  output  consistent  with  cmdr's  guidance/intent? 

Were  outputs  produced  and  distributed  in  a  timely  manner? 

■■■■ 

Was  a  “mini-AAR"  conducted  w/in  the  FSE  after  each  activity? 

Comments  (fixes/lessons  learned/critical  incidents/examples); 
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Input/Output  Flow  Among  DivArty  Elements 


Input/Output  Flow  Among  DivArty  Elements 
Fire  Support  Element  (FSE) 
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Division  Fire  Support  Element(FSE) 
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APPENDIX  C 

EVALUATION  OF  SELF  ASSESSMENT  PACKAGE  AND  MINI-AARS 


Directions:  Please  check  one  of  the  categories  of  agreement  or  disagreement  for  each  item 
below.  Abbreviate  the  name  of  your  celt  (staff  section):  OPS,  TGTG,  FSE,  FCE. 

Need/value  of  mini-AAR/self-assessment 

Date: _ Time: _ Cell  (staff  section): _ SSN  (last  4  digits) _ 


1.  The  mini-AAR/self-assessment  approach  provides  useful  training  feedbaek  to  supplement  the 
g/ven  (check  one  of  the  boxes  in  the  following  line).  ,  " 


Strongly 

Moderately 

Agree 

Disagree 

Moderately 

Strongly  Disagree 

Agree 

Agree 

Disagree 

Comments: 


2.  The  mini-AAR/self-assessment  approach  helped  improve  performance  for  me  or  my  cell  over 
the  3-day  exercise  (check  one  of  the  following  boxes).  :  ‘  ' 


Strongly 

Moderately 

Agree 

Disagree 

Moderately 

Strongly  Disagree 

Agree 

Agree 

Disagree 

Comments: 


3.  MyceRwasabletop>ickputtraim  problems  that  we  might  havo  overlooked  without  the 

(check  one  of  the  following  boxes).  -  -  ,  , 


Strongly 

Moderately 

Agree 

Disagree 

Moderately 

Strongly  Disagree 

Agree 

Agree 

Disagree 

Comments: 


C-1 


AselfiaSse. 

ssment  package 

similar  fS  this 

:  is  not  curre 

’ntly  available  to  f 

ny  unit  (check  below). 

Strongly 

Agree 

Moderately 

Agree 

Agree 

Disagree 

Moderately 

Disagree 

Strongly  Disagree 

Comments: 

i5.  A  self-assessment  package  similar  to  this 
exercises  (check  one  of  the  following  boxes 

or  an  improved  version  would  be  useful  in  future 

Strongly 

Agree 

Moderately 

Agree 

Agree 

Disagree 

Moderately 

Disagree 

Strongly  Disagree 

Comments: 

Features  of  self-assessment  methods  and  tools  (check  one  of  the  boxes  below  each  item) 

SSN  (last  4  digits) _ 


6.  Themstruction  sheet  was  clear,  well  organized,  and  had  about  the  right  amount  of 
information,  ,  ■  aS;  ':  :  f  -  *'  '  ■ 


Strongly 

Moderately 

Agree 

Disagree 

Moderately 

Strongly  Disagree 

Agree 

Agree 

Disagree 

Comments: 


?  7.  Assessment 

checklist  items; 

were  useful  as 

■memory  jo^ 

'gers  and  could  bi 

?  answered. 

Strongly 

Agree 

Moderately 

Agree 

Agree 

Disagree 

Moderately 

Disagree 

Strongly  Disagree 

Comments: 

8,  Ifie  input/output  flow  chartsrhelped  us  to  fill  out  the  forms  and  conduct  mini-AARs. 


C- 


Strongly 

Agree 

Moderately 

Agree 

Agree 

Disagree 

Moderately 

Disagree 

Strongly  Disagree 

Comments: 

9.  The  table  of  inputs^  processes,  and  outputs  helped  us  to  fill  out  the  forms  and  conduct  mini- 
AARs  .  ^  ^  ■ 


Agree 


Comments: 


Moderately 

Agree 

Disagree 

Moderately 

Agree 

Disagree 

10^  I  felt  that 

i  could  respond  1 

accurately  id 

the  self-asst 

?ssment  tooh  ' 

Strongly 

Agree 

Moderately 

Agree 

Agree 

Disagree 

Moderately 

Disagree 

Strongly  Disagree 

Comments: 

C-3 


